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APPLICATION OF HUMAN FACIAL FEATURES RECOGNITION TO AUTOMOBILE 

SECURITY AND CONVENIENCE 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This Invention relates generally to an imaging system in a vehicle for detecting the 
presence of a person and, more particularly, to an imaging system for use in a vehicle 
that identifies a human face, and uses this information for vehicle security and 
convenience purposes. 

2. Discussion of the Related Art 

The National Highway Traffic Safety Administration requires passenger side 
airbags on all passenger cars manufactured after 1998. Airbags have saved thousands 
of lives, but safer airbag deployment systems can save even more lives. Proposals have 
been submitted by the National Highway Traffic Safety Administration to require 
advanced airbags that are safer and more efficient Thus, there is a need in the art for an 
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advanced airbag deployment system that determines airbag deployment depending on 
the detection, identify and tracking of a person in the passenger seat of the vehicle. 

Future airbag deployment systems will be able to identify and track a person in the 
passenger seat of the vehicle. Based on this information, the system will provide signals 
5 whether to fire the airbag, fire the airbag at low velocity or fire the airbag at normal high 
velocity during a crash event, depending on whether a person is seated in the passenger 
seat, the size of the person and the position of the person relative to the airbag 
deployment door. In one example, if a person is detected in the passenger seat, and is 
Q some small distance (for example, within three inches) from the airbag door, then the 
J|o airbag does not fire during a crash event. If a person is detected in the passenger seat, 
\j and is close (for example, between three and eight inches) to the airbag door, the airbag 
fll is fired at a lower velocity during a crash even. If the person is detected in the passenger 
O seat, and far enough away (for example, more than eight inches) from the airbag door, 
then the airbag is fired at normal high velocity during a crash event. 

Present passenger sensing systems typically include ultrasonic, weight, infrared 
and/or electromagnetic sensors to detect a passenger for airbag deployment. Ultrasonic 
sensing systems sense the motion of objects within the passenger seat area and 
determine whether the object is moving closer or farther away from the sensor location. 
However, ultrasonic sensors don't identify the nature of the object, and thus can't tell if it 
20 is a person or some other object, such as a bag of groceries. Similar to ultrasonic 
sensors, microwave sensors employed in active doppler radar systems can track objects, 
but cannot identify human presence. Weight sensors identify objects in the passenger 
seat based on applied pressure, but don't consider the passenger's location relative to 
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the airbag. Passive IR sensors acquire thermal images of the passenger seat, but these 
systems are very costly. Present active IR sensors sense the relative location of the 
passenger relative to the airbag, but cannot identify human presence. Electromagnetic 
systems include LC resonant circuits where body capacitance is used to detect presence 
5 and identify objects, but these systems can not track the objects. 

U.S. Patent No. 5,835,613 issued to Breed et a!., November 10, 1998, discloses a 
vehicle interior monitoring system that claims to identify, locate and monitor persons in the 
passenger compartment of the vehicle. The monitoring system employs infrared emitters 
C3 that illuminate the interior of the vehicle, and charge couple device (CCD) arrays that 
3:0 detect the radiation. Outputs from the CCD arrays are analyzed by computational 
devices that employ pattern recognition algorithms to classify, identify or locate the 
fii content or objects in the passenger seat. The pattern recognition system for determining 
Q vehicle occupants disclosed in the '613 patent employs complicated software that must 
fU learn the shape of an individual in all kinds of lighting situations under various conditions. 
|S Additionally, employing pattern recognition in this manner is limited in its ability to track 
the individual as he or she moves around in the passenger seat. Further, the ability to 
identify and track humans by general pattern recognition is questionably unreliable. 
Pattern recognition cannot identify who the person is, only detect an object's shape. 

What is needed is an improved airbag deployment sensor and system that is able 
20 to effectively identify and track a person in the passenger seat of a vehicle. It is therefore 
an object of the present invention to provide such a sensing system for vehicle passenger 
airbag deployment 
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SUMMARY OF THE INVENTION 

In accordance with the teachings of the present invention, an imaging system for 
providing vehicle security and convenience features is disclosed that employs face 
recognition software and algorithms to identify and track a person. The system includes 
infrared emitters that emit an infrared signal along a predetermined field-of-view, and an 
infrared sensor, such as a CMOS sensor used as a video signal array, that receives 
reflected infrared illumination from objects in the field-of-view. A processor, including the 
face recognition software, is employed to detect human faces. Once a face is detected, it 
can be compared to a data base to identify the person. 

Various applications of the imaging system for providing driver convenience and 
security include determining driver Identification as the driver approaches the vehicle for 
automatically unlocking the door and starting the vehicle; determining if a potential thief is 
in the vehicle by face recognition; and providing driver seat adjustment, rear and side 
mirror adjustment and steering wheel adjustment. 

Additional objects, advantages and features of the present invention will become 
apparent from the following description and appended claims, taken in conjunction with 
the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a side, cut-away, plan view of a person in the passenger seat of a 

vehicle in connection with an image sensing system for airbag deployment, according to 

an embodiment of the present invention; 

Figure 2 is a perspective view of a video camera and LED illuminator unit 

employed in the airbag deployment system of the present invention; 
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Figure 3 is a graph with wavelength on the horizontal axis and luminous energy on 
the vertical axis showing the luminance curve for sunlight; 

Figure 4 is a representation of camera orientation and position parameters with 
respect to the center and normal of the airbag door; 

Figure 5 is a block diagram of the airbag deployment system of the present 
invention; 

Figure 6 is a schematic diagram depicting a frame differencing technique used in 
the airbag deployment system of the present invention; 

Figure 7 is a timing diagram for the frame differencing technique of the invention; 

Figure 8 is a top plan view of a vehicle showing various sensor locations for 
sensing different areas of the vehicle for various security and convenience purposes, 
according to an embodiment of the present invention; 

Figure 9 is a side view of a vehicle showing a camera having a field-of-view for 
detecting objects in front of the vehicle, according to another embodiment of the present 
invention; 

Figure 10 is a top view of a vehicle including a side camera and a rear center 
camera for detecting objects at the side and rear of the vehicle, according to an 
embodiment of the present invention; 

Figure 1 1 is a side view of a vehicle including a camera positioned in front of the 
driver of the vehicle, according to another embodiment of the present invention; and 

Figure 12 is a side view of a vehicle including a camera mounted in the dome light 
of the vehicle for intrusion purposes, according to another embodiment of the present 
invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The following discussion of the preferred embodiments directed to an airbag 
deployment system that employs human facial feature recognition is merely exemplary in 
nature, and is in no way intended to limit the invention or its applications or uses. 

According to one embodiment of the present invention, a technique for detecting, 
identifying and tracking a person in the passenger seat of a vehicle is disclosed. As 
discussed above, it is sometimes desirable to deploy a passenger side airbag in a vehicle 
at low velocity, or not deploy the airbag, during a crash event, depending on whether a 
person is occupying the passenger seat of the vehicle, the proximity of the person to the 
airbag door, and the size of the person. According to the present invention, the 
identification and tracking of the person is determined by face recognition software, in 
particular, software that recognizes and tracks the person's eyes and other facial 
features. Tracking can take place that way at many head angles and poses. The 
software algorithm would use calibrated face and feature separation to estimate range. 
The tracking of a localized area on the human face allows more image frames to be 
acquired per second, permitting more frequent tracking of the person. The more frequent 
tracking of the facial features will prevent acquiring a blurry image because the image is 
updated more frequently. 

Various software is known in the art that processes data from video data patterns 
received from an object being analyzed, and determines whether the object has a face. • 
For example, such software includes the Visionics Face-It software, well known to those 
skilled in the art. It is stressed that the present invention is not limited to any particular 
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facial feature mapping function, but can include any known algorithm, suitable for the 
purposes described herein, for recognizing facial features, whether it be two-dimensional 
or three-dimensional, that are then also to be used for ranging functions, as well. Further, 
according to the present invention, ranging algorithms are used in combination with the 
5 know face recognition software. 

As will be discussed in detail below, the present invention employs infrared 
radiation reflected off of objects in the passenger side of the passenger compartment of a 
vehicle that is received by an electronic video camera. The video camera generates the 

Q electrical signals and image used by the face recognition software to determine the 

JfJO presence identify and tracking of the person. 

G Figure 1 is a cut-away, side, plan view of the passenger side compartment 10 of a 

fi] vehicle 12. In this depiction, a person 16 is shown in the passenger seat 18, where the 
C3 person 16 is moving forward during a crash event. An airbag pillow 20 is shown being 
fy deployed through an airbag door 24 housed in an instrument panel 22 during the crash 
JI55 event. 

A video camera and IR LED illuminator unit 26 is mounted on the instrument panel 
22 at a location suitable for the purposes described herein. Figure 2 is a perspective view 
of the unit 26 removed from the vehicle 12. The unit 26 includes a cluster 28 of IR LEDs 
30. A plurality of LEDs 30 is provided to generate the necessary intensity for daylight 
20 operation. The cluster 28 emits a beam of IR radiation towards the person 16 that is 
reflected therefrom back towards the unit 26. A video camera 34 is provided in the unit 
26 to receive the reflected radiation from the person 16. The video camera 34 is used by 



7 



TRW Docket No. 12-1134 

way of a non-limiting example in that any detector that detects infrared radiation suitable 
for the purposes described herein can be used. 

A filter 36 is provided over the camera 34 to filter out radiation that is not within the 
desirable infrared range. The filter 36 can be any filter suitable for the purposes 
5 described herein, such as a Ti02 filter or a polarizing filter. The filter layers and thickness 
can be chosen to transmit the IR image to the detector, but to reflect the visible image 
away from the detector. The polarizing filter can be used to reduce visible light to the 
detector using electro-optical polarization that passes the IR wavelengths, but strongly 
attenuates the non-IR wavelengths. Figure 3 shows the luminance curve for sunlight, 
TO where the filter 36 passes infrared radiation in the 40 nm bandpass window. The filter 36 
NJ provides some protection against sunlight that may affect the operation of the airbag 

deployment system, and recognition of the face of the person 16. 
rj In this embodiment of the present invention, a single camera is used to acquire 

U and monitor the range of the person 16. The software employed to perform this function 
utilizes two separate locations on the occupant's face to provide the ranging. In a 
preferred embodiment, it is the person's eyes that are detected to provide the 
triangulation for ranging purposes. However, as will be appreciated by those skilled in the 
art, other facial features of the person 16 can also be used, such as the person's ears, 
etc. Also, the software algorithm allows the person's head size to be determined so that 
20 both eyes do not need to be in view to track the person after he has been acquired. 
Additionally, the software can be used to view other parts of the person's body, such as 
the person's torso, in combination with the facial feature or head size detection. 
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Because human facial features are unique, a database can be used to store 
specific information, such as eye-to-eye separation, about a person, so that the software 
can particularly identify that person. This is important, in one example, so that the system 
can identify children and fifth-percent females, and inhibit air bag firing for these people, 
5 as required by government mandated regulations. Also, being able to specifically identify 
a person improves the ranging accuracy of the system because the system knows that 
person's eye separation or other specific facial features. 

The acquisition and tracking software needs to be calibrated for a particular seat 
^% position, airbag position, vehicle makeup, etc. Figure 4 is a representation of the camera 
rto orientation and position parameters with respect to the center and normal of the airbag 
door 24. The three parameters that must be estimated in the vehicle for ranging 
W according to the invention after the camera and Visionics Face-It software are calibrated 
^ include two position offsets and one angular offset. The two position offsets are the 
lateral offeet of the camera 34 with respect to the center of the airbag (AY), measured 
|j5 laterally and perpendicularly to the normal vector out of the center of the airbag door 24, 
and the front or back offset (ziX) of the camera 34 with respect to the center of the airbag 
door 24, measured along an axis parallel to the normal out of the center of the airbag 
door 24. The angular parameter is the azimuth offset (AG) between the optical axis of the 
camera 34 and the normal vector coming out of the center of the airbag 20. Only two of 
20 these parameters, AX and AO, are used in the modified monocular distance equation. 

A calibration procedure, described below, can be used to determine AY, AX and 
AO. There is a trade off between measuring a large number of calibration points and 
linearly interpolating between them or measuring a small number of calibration points and 
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non-linearly interpolating between them. In theory, calibrating with a large number of 
points would not require a model for estimating that is described herein. The cost of 
making the large number of measurements necessary for a brute force calibration and the 
possibility that out of range operation of the camera 34 or software might occur unnoticed 
with brute force calibration must be considered before using that approach. The 
approach used herein uses a small number of calibration tests in order to form a model 
for interpolation. 

From Figure 4, the following equations can be written in order to relate 
measurements made in the airbag door reference frame (xyz) to those made in the 
camera lens reference frame (xyz). Only face positions in front of the airbag door (x>o) 
are considered in the following equations. For p,0 (pitch or elevation angle) and ^ (yaw or 
azimuth angle) defined in the camera's spherical coordinate system, the equations for 
displacements x and y in the airbag coordinate system are: 



This assumes that 9 + AG is fixed during the calibration. Rearranging equations (1) and 
(2) yields: 



x=-AX+psin(e+Ae)cos((t)+A(|)) 



(1) 



y=-AY+psin(e+Ae)sin(<j)+A(()) 



(2) 



y + AY 
x + AK 



= tan (^^ + AO) 



(3) 
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Making eye coordinate readout measurements at fixed x+AX, and then taking the slope of 
the tangent with respect to changes In y yields: 



1 



x + AX 



fixed _x - 



d tan(^ + AO) 

dy 



(4) 



Knowing x and the result on the right side of equation (4), AX can be determined. 
Knowing AX, equation (3) and the measured data can be used to determine AY. Then, 
using AX and AY, equation (3) and the data, Acj) can be determined. Using equation (3), 
and the fact that: 



eye_separation_distance=yrt eye-yi 



eye yieft_eye, 



(5) 



15 



the modified monocular equation follows from equation (6) for defining the objective 
parameter x, or the eye to airbag distance. 



x = -AX + 



eye _ separation _ dis tan ce 



tan(4/, _ + AO) - tan(^,, _ + AO) 



(6) 



20 



The calibration table consists of a lookup table of the SDK determined eye 
coordinate as a pixel value linked to the associated ray slope, as viewed in the camera 
coordinate system. Equation (6) can be simplified in terms of ray slopes (tangents of 
singular angles), so that the calibration table can be used directly to determine the eye to 
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airbag distance from monocular operation, 
equation (7) below, 
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Employing the tangent trigonometry identity in 



^ . tan(a) + tan(6) ,„ 

tan(a + 6) = — ^ (7) 

l-tan(a)tan(b) 



and applying this identify to equation (6), leaves the modified monocular equation in a 
form that can be used by direct look up of tangent/slopes from the calibration lookup 
table. This equation is given in equation (8) below. 



x = -AX + 



eye _ separation _ dis tan ce 



tan(^/./,_.^.) + tan(AO) tan(^,,_,^,) + tan(A(D) 



(8) 



1 - tan(^;,^, ) tan(A<D) 1 - tan($J,,_,^, ) tan(AO) 



Figure 5 is a block diagram of an imaging system 50 of the invention including a 
digital signal processor (DSP) 52. The DSP 52 includes the face recognition software 
and ranging functions that performs the analysis on the image generated by the camera 
34. A clock generator 60 provides timing for the various digital devices in the system 50, 
and a power management system 62 provides the power. The DSP 52 is connected to a 
CMOS chip 70 including a pixel array 72 that represents the IR detector, such as the 
camera 34. In this example, the pixel array 72 includes 256x256 pixels to provide the 
desired level of resolution. The CMOS chip 70 also includes various elements for timing 
and control purposes, including state matching logic 74, clock/timing circuitry 76, analog 
conditioning circuit 78, registers/buffers 80, on-chip programmable logic circuitry 82, etc. 
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Additionally, an analog-to-digital converter 84 is also provided to convert the analog signal 
from the pixel array 72 to a representative digital signal. An on-chip SRAM memory 86 is 
shown for storage purposes, but can be off-chip as well. The operation of these devices 
in the system described herein would be apparent to those skilled in the art. 

In one embodiment, the infrared LEDs 30 are continuously on to provide the 
reflected radiation received by the camera 34. However, typically some kind of filtering or 
signal processing must be done to correct for the problems caused by direct sunlight on 
the camera 34 that make it through the filter 36. Particularly, the system 50 needs to be 
able to distinguish between shadows caused by sunlight and actual edges of features on 
the occupant 16. According to the present invention, a frame differencing technique is 
employed that synchronously pulses the LEDs 30 on for a predetermined period of time 
and a predetermined number of frames of video data, and then off for a predetermined 
period of time over the same number of frames of video data. Then, the frames of data 
are subtracted from each other so that the frames without IR illumination can be 
subtracted from the frames with IR illumination, and the background can be eliminated. 
The detector is electronically shuttered synchronously with the pulses to provide exposure 
control. The frame differencing technique described herein is used in conjunction with the 
infrared pulsing to achieve the desired result. In other words, the frame differencing is 
synchronized to the infrared pulses. 

The concept of frame differencing is the time-aperture, pixel-level storage of 
images using natural illumination only and natural illumination with added infrared 
illumination. Frame differencing allows these images to be subtracted to mitigate the 
effects of strong visible illumination. The set-up includes a neutral density filter that sets 
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the IR illumination, plus worst case background to maximize analog-to-digital converter 
input. Face recognition requires that the worst case analog-to-digitai range for the 
differenced image be 5 to 6 bits. The visible light would fit within the remaining range 
allowed by the analog-to-digital converter. The image differencing is either performed in 
the analog domain, where two pixel level capacitors are charged, one at each illumination 
level, or in the digital domain where a RAM memory of the digitized pixel output is taken 
at each illumination. The frame differencing acts to subtract out the illumination effects of 
the visible illumination, and to improve the image contrast. The frame differencing 
function can be performed in the acquisition/high bandwidth generation mode or in the 
narrow bandwidth track mode using pulsed LED illumination. The number of the 
electrons from the pulsed IR light source must be 10 times greater than the photon noise 
of the ambient illumination. Here, the noise of the ambient illumination is the square root 
of two times the number of electrons within the sun intensity, because two image frames 
are being acquired for every one IR image received. 

Figure 6 is a representation of how the frame differencing is performed in the 
camera 34, according to one embodiment of the present invention. Figure 7 is a signal 
timing line showing the operation of the frame differencing technique of the invention. A 
pixel array 90 in the camera 34 receives radiation from the scene for a predetermined 
time period (10 [is) during a pulse of IR from the LEDs 30. At this time, the pixel array 90 
receives ambient light and infrared light. The charge stored by each pixel or photodiode 
92 in the array 90 is then transferred to a charge storage site 94 made up of a plurality of 
capacitors, one capacitor 96 for each pixel 92. About 10 us later, at the end of the pulse 
from the cluster 28, the pixel array 90 detects just the ambient light for the same time 
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period. The charge received by the pixel array 90 at this time period is stored in a 
capacitor storage site 98, having capacitors 100. An electronic shutter is employed in the 
detector to open and close at the appropriate time synchronously with the pulses of IR 
radiation for the operation described herein. 

The two storage sites 94 and 98 are summed In a summation amplifier 102. The 
difference between the two storage sites 94 and 98 is then digitized by an analog-to- 
digital converter 104, and represents the frame of data where the ambient light has been 
removed. The readout of data talces about 10 ms, then at the next time period, the next 
pulse from the cluster 28 occurs. The complete frame differencing process can be 
performed on a single chip in CMOS where the pixel array 90 and the storage sites 94 
and 98 are together. In an alternate embodiment, the frame differencing is performed at 
a different time period at an off-chip site, where the storage sites 94 and 98 are RAM. 

The frame differencing technique of the invention can be described in the following 
manner. The variables are defined as ambient illumination l(x,y), direct ambient 
illumination T(x,y), scene reflectance R(x, y), and modulated source as L when on, and 0 
when off. The response of the camera 34 is proportional to the product of reflectance and 
illumination. 

S(x,y,OFF) = k*(l(x,y)*R(x,y))+T(x,y) 
S*(x,y,ON) = k*((L+l(x,y)rR(x,y))+T(x,y) 
D(x,y) = S(x,y,ON) - S(x,y,OFF) = KL*R(x,y) 
This difference scene has much smaller dynamic range than the simple image 
S(x,y,OFF). The same benefit can be derived by reading out the frame with LED, then 
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reading the frame without LED and subtracting the frames external to the camera 34. 
The penalty is increased dynamic range required to avoid saturation. 

L must be much greater than the direct ambient illumination response I. Thus, I is 
made as small as possible by using a narrow bandpass filter aligned in frequency to L. 
5 The raw sampling rate has to be twice the requirement set by the object tracking because 
two frames are differenced to get one frame to supply the face recognition software. The 
LEDs 30 have to be much faster. The IR radiation source has to be modulated such that 
all emission is during the time when all detectors are active. If the integration times of all 

O pixels in the detector are not aligned, the available time for the source to be ON is 

II) reduced by the worst case misalignment. 

[j The face recognition system described above for identifying and tracking a person 

ru in a vehicle passenger seat can be extended to other types of vehicle systems for other 
O purposes. Figure 8 is a top plan view of a vehicle 110 showing sensors or cameras 
W positioned at various locations for other purposes. For example, a forward looking 
infrared sensor 112 is positioned at a center location on the instrument panel 22 and is 
directed fonA/ard for detecting objects in front of the vehicle. Additionally, a rearward 
looking infrared sensor 114 is positioned at a center location on the instrument panel 22 
and is directed towards the rear passenger compartment of the vehicle 110 to monitor 
activity in that part of the vehicle. Additionally, an infrared sensor 1 16 is positioned on the 
20 instrument panel 22 in front of the driver seat 1 18 to detect the driver's facial features for 
various purposes discussed below. An infrared sensor 120 is positioned on the 
instrument panel 22 in front of the passenger seat 122 for airbag deployment as 
discussed above. The various sensors 112, 114, 116 and 120 can be used in different 
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ways in different systems. Other sensor locations can also be employed for other 
purposes consistent with the discussion herein. Some of the applications are discussed 
below. 

In one embodiment, according to the present invention, a sensor or camera is 
pointed towards the passenger and driver's seat that takes an image of the driver and the 
passenger's facial feature. Using the facial feature software, the contours of the face and 
eye coordinates of the driver and the passenger are compared to images stored in a 
computer data base or lookup table. If the software makes a match of the human face 
with one stored in the data base, the driver and/or passenger airbag will be adjusted so 
that that person's characteristics are used to personalize airbag deployment. This will 
help the airbag to automatically adjust for low weight, young persons, short drivers, etc. 

According to the invention, a drowsy driver application involves identifying when 
the driver is falling asleep at the wheel and waking the driver up. Initially, at regular time 
intervals, the face recognition software acquires the facial features such as eyes, or any 
combination of 2-D or 3-D facial/head/torso feature coordinates of the driver. Depending 
on the new coordinates of the driver's facial features, the software will send a warning to 
the driver that he or she is falling asleep. This warning will be based on the frequency of 
the driver's head nods, i.e., the back and forth head motion, whether the driver's eyes are 
closed, etc. In particular, the percentage of time that the eyes are closed during a fixed 
time interval is used to determine if the driver is asleep. Preliminary studies have shown 
that when a driver's eyes are shut more than 70% of the time over two seconds, the 
driver is drowsy and may be falling asleep. In addition, the back and forth motion of the 
driver's head or the driver's eyes looking at a cell phone will be utilized to determine if the 
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driver is paying attention to the road ahead. The face recognition software can sense the 
difference in the driver's eye coordinates and this information can be relayed back to the 
driver. This information returns to the driver in the form of a mechanical motion, or an 
electrical stimulus or the production of a loud signal to wake the driver. 

Similar to the drowsy driver application discussed above, drug use could cause the 
driver to fall asleep. Presently, many drivers use caffeine or speed to keep them awake 
over a long trip. However, when the drug wears off, the driver's response time diminishes 
to critical changes in road or driving conditions. Thus, monitoring of the driver's pupils will 
help detect a driver using drugs when the drugs stop working. Basically, the facial feature 
recognition software determines how often the iris of the eye is blocked by the driver's 
eyelid to determine whether the driver is failing asleep. 

The facial feature recognition system described herein can also be used for driver 
identification. When the driver enters the vehicle, a camera pointing outward placed near 
the driver's door will take an image of the driver's facial features. Using the facial feature 
recognition software, the contours of the facial features such as eyes, or any combination 
of 2-D or 3-D facial/head/torso feature coordinates of the perspective driver are compared 
to images stored in the software data base. If the software makes a match of the face 
with one stored in the data base, the doors will open and the vehicle can be automatically 
started. OthenA^ise, the vehicle will remain in the off position. In an alternate 
embodiment, after the driver gains access to the vehicle and is in the driver seat, the 
camera can be used to identify the driver for purposes of starting the vehicle. 
Additionally, in a parental control situation, the driver can be identified as a young person. 
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where the algorithm sets the control of the vehicle to a maximum speed for that driver's 
use. 

The driver ID concept can be extended to driver convenience features using the 
face recognition software as well. Particularly, the driver's human facial features can be 
used to recall the driver's last seat position, rear and side mirror adjustment, steering 
wheel setting, radio setting, etc. In this example, the vehicle's computer can store the 
driver's last seat position, and rear and side mirror adjustment with the driver's human 
facial features. Subsequently, whenever the camera acquires the driver's facial features 
and this Image matches a stored driver's personal preference, the seat and mirrors can 
be automatically repositioned to the desired locations. 

The use of the specific imaging and processing techniques of the present invention 
allows the presence detection of people of all ages and sizes, whether conscious or 
asleep, in the interior of a vehicle with very high reliability. This detection is effective at all 
times, for stationary or moving vehicles, whether the engine is running or not, without 
being perturbed by external stimulus to the vehicle activities so that determination of 
dangerous human interior situations Is enabled to generate warning alerts locally or to be 
communicated. This Includes situations of abandoned children, medical emergency, 
captive occupants, sleeping occupants, etc. 

Figure 9 is a side view of a vehicle 130 Including a fonwarded mounted sensor 132 
that provides a sensor fleld-of-view In front of the vehicle 130. Reflections from objects In 
the forward looking direction, such as a pedestrian in a crosswalk, a slow moving vehicle, 
etc., will be registered by the processing system, and provide a warning to the driver of an 
object in the vehicle's path. By recognizing a pedestrian in a crosswalk or the like, pre- 
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crash avoidance can be provided using the face recognition system of the present 
invention. 

Figure 10 is a top view of a vehicle 140 that includes a center mounted camera 
142 behind the vehicle 140 for providing an indication of objects in the vehicle's path 
when it is moving in a rean/vard direction. Additionally, a camera 144 placed in the 
driver's side mirror 146 can detect objects located in the driver's blind spot to provide 
warnings that the driver may not see. In this example, the camera 144 will allow the 
driver to stay focused on the road ahead, but can still safely make a right or left turn or 
lane change. The camera 144 can be provided for detecting pedestrians or riders on 
bicycles in the vehicle blind spot by the face recognition system. 

In severe automobile collisions, it is sometimes difficult to determine what caused 
the accident. In one example, before the crash event, the exterior forward looking 
camera can make pre-crash images. These images will be relayed back to the driver in 
the form of an electrical or mechanical warning so that the driver can make adjustments 
to avoid a collision. In the event of a vehicle collision, the images acquired will be stored 
so at a later date, the pre-crash sensing conditions will be available to make a post crash 
injury assessment. After the crash, a crash notification signal can be automatically 
transmitted in the form of a radio wave signal which will alert the police and fire 
department of the crash location. 

To keep the driver's attention on the road and hands on the steering wheel, the 
driver's facial gestures, such as movement of facial features, can be used to perform 
tasks previously performed manually. For example, a driver's facial feature command 
can be compared to an existing stored facial feature that will cause the vehicle to 
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automatically change the automobile controls without the driver taking his or her eyes off 
the road. Some examples include control of automobile accessories, such as turning on 
and off the radio, turning on and off the vehicle lights, turning on and off the windshield 
wipers or blowing the horn. Figure 11 shows a side view of a vehicle 152 including a 
5 sensor 154 of the type discussed herein positioned in front of a driver 156 for identifying 
and tracking the facial features of the driver 156. The face recognition system of the 
invention can use this information in the manner as discussed herein. 

Further, a camera can be positioned in the vehicle to monitor intrusion. In this 
^i. example, a camera takes an image anywhere inside of the vehicle passenger 
3iO compartment. If a person is detected in the backseat of the vehicle, the camera will take 
\| an image of the person's facial features. If the facial features match one stored in the 
rU computer, then no alarm will go off. Otherwise, an alarm can be sounded to alert the 
O driver that someone is in the back seat. The camera can provide a recording of the 
Jj:; intruder for later identification, and provide detection of human presence after the vehicle 
flip is locked. Figure 12 shows a side view of a vehicle 160 depicting this example, where a 
sensor 162 is mounted at a center location in the vehicle passenger compartment, such 
as in the center dome light. 

The foregoing discussion discloses and describes merely exemplary embodiments 
of the present invention. One skilled in the art will readily recognize from such discussion 
20 and from the accompanying drawings and claims, that various changes, modifications 
and variations can be made therein without departing from the spirit and scope of the 
invention as defined in the following claims. 
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CLAIMS 

What is Claimed is: 

1 1. An imaging system for a vehicle, said system comprising: 

2 an infrared source generating an infrared beam along a particular field-of- 

3 view in or around the vehicle; 

4 an infrared detector responsive to infrared radiation reflected from objects 

5 in the field-of-view of the infrared beam, said detector generating image signals from 
C:6 the reflected infrared radiation; and 

j[f a processor responsive to the image signals from the detector, said 

;j processor including face recognition software that acquires and tracks facial features of 

ff a person from the image signals, said processor causing a vehicle operation to be 

^ performed if a person's face is detected. 

y 2. The system according to claim 1 wherein the processor compares a 

2 detected image of a person's face with stored images to identify the person. 

1 3. The system according to claim 2 wherein the system is a driver 

2 identification system, said processor identifying a driver's face and causing a vehicle 

3 door to automatically unlock if the driver's face is detected. 
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4. The system according to claim 2 wherein the system is a driver 
identification system, said system identifiying a driver's face, said system monitoring 
driver gestures to perform a particular vehicle function if the driver's face is detected. 

5. The system according to claim 1 wherein the system is a vehicle 
convenience system, said processor identifying a person's face and causing a vehicle 
system to be automatically adjusted in response to detecting the person's face. 

6. The system according to claim 2 wherein the system records the person's 
face if it does not match a face stored in the processor. 

7. The system according to claim 2 wherein the processor causes the 
vehicle to start if the detected image matches a particular stored image. 

8. The system according to claim 2 wherein the processor controls the 
vehicle speed if a detected image is of a particular person's face that is stored. 

9. The system according to claim 1 wherein the processor tracks and 
monitors the persons eyes. 

10. The system according to claim 1 further comprising an infrared filter 
positioned in front of the detector. 
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11. The system according to claim 1 wherein the infrared detector includes a 
pixel array of photodiodes. 

12. The system according to claim 1 wherein the infrared source is pulsed on 
for a predetermined time at predetermined intervals. 

13. The system according to claim 12 wherein the pulsed source and the 
detector are synchronized such that a difference of successive frames represents an 
image free of ambient light variations. 

14. The system according to claim 1 wherein the detector includes a pixel 
array of diodes. 

15. A driver imaging and identification system for use in a vehicle, said 
system comprising: 

an infrared source generating an infrared beam along a particular field-of- 
view in or around the vehicle, wherein the infrared source is pulsed on and off at 
predetermined intervals; 

an infrared detector responsive to infrared radiation reflected from objects 
in the field-of-view of the infrared beam, said detector generating image signals from 
the reflected radiation; and 

a processor responsive to the image signals from the detector, said 
processor including face recognition software that acquires and tracks facial features of 
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11 a person from the image signals and compares a detected image of a person's face 

12 with stored images to identify the person, said processor identifying a driver's face from 

13 the comparison and causing a vehicle door to automatically unlock if the driver's face is 

14 detected. 

1 16. An intruder detection system for use in a vehicle, said system comprising: 

2 an infrared source generating an infrared beam along a particular field-of- 

3 view in or around the vehicle, wherein the infrared source is pulsed on and off at 
predetermined intervals; 

^ijs an infrared detector responsive to infrared radiation reflected from objects in 

SjS the field-of-view of the infrared beam, said detector generating image signals from the 

fy7 reflected radiation; and 

CB a processor responsive to the image signals from the detector, said 



processor including face recognition software that acquires and tracks facial features of a 
person from the image signals and compares a detected image of a person's face with 

11 stored images to identify the person, wherein if the identified face does not match a 

1 2 stored image, the processor causes an alarm to sound. 



1 17. The system according to claim 16 wherein the processor provides a 

2 recording of the image if the detected image does not match a stored image. 

1 1 8. A driver convenience system for use in a vehicle, said system comprising: 

2 an infrared source generating an infrared beam along a particular field-of- 
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3 view in or around the vehicle, wherein the infrared source is pulsed on and off at 

4 predetermined intervals; 

5 an infrared detector responsive to infrared radiation reflected from objects in 

6 the field-of-view in or around the vehicle; 

7 an infrared detector responsive to infrared radiation reflected from objects in 

8 the field-of-view of the infrared beam, said detector generating image signals from the 

9 reflected radiation; and 

10 a processor responsive to the image signals from the detector, said 
processor including face recognition software that acquires and tracks facial features of a 

IJ2 person from the image signals and compares a detected image of a person's face with 

<!j3 stored images to identify the person, said processor causing a vehicle system to be 

rii4 automatically set in response to detecting the person's face. 



J^H 19. A driver imaging and identification system for use in a vehicle, said 
system comprising: 

3 an infrared source generating an infrared beam along a particular field-of- 

4 view in or around the vehicle, wherein the infrared source is pulsed on and off at 

5 predetermined intervals; 

6 an infrared detector responsive to infrared radiation reflected from objects 

7 in the field-of-view of the infrared beam, said detector generating image signals from 

8 the reflected radiation; and 

9 a processor responsive to the image signals from the detector, said 
10 processor including face recognition software that acquires and tracks facial features of 
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11 a person from the image signals and compares a detected image of a person's face 

12 with stored images to identify the person, said processor identifying a driver's face from 

13 the comparison and monitoring driver commands to perform a particular vehicle 

14 function. 



1 20. The system according to claim 1 9 wherein the processor detects the person 

2 after the vehicle is locked. 

01 21 . A driver convenience system for use in a vehicle, said system comprising: 

|j)2 an infrared source generating an infrared beam along a particular field-o^ 

CJ3 view in or around the vehicle, wherein the infrared source is pulsed on and off at 

flj4 predetermined intervals; 

C35 an infrared detector responsive to infrared radiation reflected from objects in 

riB the field-of-view in or around the vehicle; 

y[7 an infrared detector responsive to infrared radiation reflected from objects in 

8 the field-of-view of the infrared beam, said detector generating image signals from the 

9 reflected radiation; and 

10 a processor responsive to the image signals from the detector, said 

1 1 processor including face recognition software that acquires and tracks facial features of a 

12 person from the image signals and compares a detected image of a person's face with 

13 stored images to identify the person, said processor causing the vehicle to start if the 

14 detected image is determined to be the vehicle driver. 
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22. A driver convenience system for use in a vehicle, said system comprising: 

an infrared source generating an infrared beam along a particular field-of- 
view in or around the vehicle, wherein the infrared source is pulsed on and off at 
predetermined intervals; 

an infrared detector responsive to infrared radiation reflected from objects in 
the field-of-view in or around the vehicle; 

an infrared detector responsive to infrared radiation reflected from objects in 
the field-of-view of the infrared beam, said detector generating image signals from the 
reflected radiation; and 

a processor responsive to the image signals from the detector, said 
processor including face recognition software that acquires and tracks facial features of a 
person from the image signals and compares a detected image of a person's face with 
stored images to identify the person, said processor causing the vehicle speed to be 
controlled in response to detecting a particular person. 
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APPLICATION OF HUMAN FACIAL FEATURES RECOGNITION TO 
AUTOMOBILE SECURITY AND CONVENIENCE 

ABSTRACT OF THE DISCLOSURE 

An imaging system (50) for providing veliicle security and convenience features 

that employs face recognition software to identify and track a person. Tlie system (50) 

employs infrared emitters (30) that emit an infrared signal along a predetermined field-of- 

view, and an infrared sensor (34), such as a CMOS sensor used as a video signal array, 

that receives reflected infrared illumination from objects in the field-of-view. A processor 

(52) including the face recognition software, is employed to detect human faces to identify 

and track the person. Once a face is detected, it can be compared to a data base to 

identify the person. Various applications for the imaging system (50) to provide driver 

convenience and security include determining driver identification as the driver 

approaches the vehicle, detemnining if a potential thief is in the vehicle by face 

recognition, providing driver seat adjustment, rear and side mirror adjustment and 

steering wheel adjustment, providing vehicle speed control, automatically starting the 

vehicle, etc. 
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